Skip to content

Fix img2img DDIM index out of bound #1137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
lstein merged 1 commit into from
Oct 18, 2022
Merged

Fix img2img DDIM index out of bound #1137

lstein merged 1 commit into from
Oct 18, 2022

Conversation

@wfng92
Copy link
Contributor

@wfng92 wfng92 commented Oct 17, 2022 •
edited
Loading

Added a community solution to fix index out of bound when doing img2img generation with ddim sampler. Also, restored steps_out to be ddim_timesteps + 1 since the removal was meant to fix the 1000 steps issue

@wfng92
Fix img2img DDIM index out of bound
fcb1858 
Added a [community solution](CompVis/stable-diffusion#111 (comment)) to fix index out of bound when doing img2img generation with `ddim` sampler. Also, restored `steps_out` to be `ddim_timesteps + 1` since the removal was meant to fix the [1000 steps issue](CompVis/stable-diffusion#111)
@lstein
Copy link
Collaborator

lstein commented Oct 17, 2022

Thanks for this. Is there a test case that you use to test the bug before and after the fix? It is certain values of step, is that right?

@wfng92
Copy link
Contributor Author

wfng92 commented Oct 18, 2022 •
edited
Loading

Thanks for this. Is there a test case that you use to test the bug before and after the fix? It is certain values of step, is that right?

After merging version 2.0 to my local machine, I encountered the following error frequently during img2img generation:

../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:145: operator(): block: [0,0,0], thread: [0,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.

Traceback (most recent call last):
  File "/stable-diffusion/./ldm/generate.py", line 388, in prompt2image
    results = generator.generate(
  File "/stable-diffusion/./ldm/invoke/generator/base.py", line 78, in generate
    image = make_image(x_T)
  File "/stable-diffusion/./ldm/invoke/generator/img2img.py", line 45, in make_image
    samples = sampler.decode(
  File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/stable-diffusion/./ldm/models/diffusion/sampler.py", line 347, in decode
    outs = self.p_sample(
  File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/stable-diffusion/./ldm/models/diffusion/ddim.py", line 79, in p_sample
    sigma_t = torch.full((b, 1, 1, 1), sigmas[index], device=device)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Image generation no longer work after that and restart is required. The common configuration that I used at that time is as follows:

  • sampler_name: DDIM
  • steps: 12
  • cfg_scale: 0.65 (I used other values as well between 0.01 to 0.95)

After applying the fix, I no longer receive the error above. Please note that the value of ddim_timesteps is not the same.

# Step 12 (before)
[  0  83 166 249 332 415 498 581 664 747 830 913 996]
# Step 12 (after)
[  0  83 166 249 332 415 498 581 664 747 830 913]

# Step 18 (before)
[  0  55 110 165 220 275 330 385 440 495 550 605 660 715 770 825 880 935 990]
# Step 18 (after)
[  0  55 110 165 220 275 330 385 440 495 550 605 660 715 770 825 880 935]

# Step 24 (before)
[  0  41  82 123 164 205 246 287 328 369 410 451 492 533 574 615 656 697 738 779 820 861 902 943 984]
# Step 24 (after)
[  0  41  82 123 164 205 246 287 328 369 410 451 492 533 574 615 656 697 738 779 820 861 902 943]

# Step 27 (before)
[  0  37  74 111 148 185 222 259 296 333 370 407 444 481 518 555 592 629 666 703 740 777 814 851 888 925 962 999]
# Step 27 (after)
[  0  37  74 111 148 185 222 259 296 333 370 407 444 481 518 555 592 629 666 703 740 777 814 851 888 925 962]

It seems that the error only happens when num_ddim_timesteps % 3 == 0 (step is multiple of 3). For example, using the step 27 will result in out of bound when using the original steps_out = ddim_timesteps + 1 code. This fix will resolve both issue as it excludes the extra time step in the bad cases.

@lstein
Copy link
Collaborator

lstein commented Oct 18, 2022

That would explain why I never see the error. I tend to use steps of 50, 20 and 10 during testing.

I'll go ahead and approve.

lstein
lstein approved these changes Oct 18, 2022
View reviewed changes
Copy link
Collaborator

@lstein lstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lstein lstein merged commit 1c2bd27 into invoke-ai:development Oct 18, 2022
@wfng92 wfng92 deleted the patch-2 branch October 21, 2022 05:11
This was referenced Oct 27, 2022
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lstein lstein lstein approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wfng92 @lstein